Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.
You Can Build A Data Pipeline In <90 Min.
Extract. Transform. Read.A newsletter from Pipeline: Your Data Engineering ResourceHi past, present and future data professional! Since today is a U.S. holiday, I won’t take much of your time; the good news is that, when conducted efficiently, building a data pipeline doesn’t have to take days, weeks or months. In fact, you can build a data pipeline in as little as 90 minutes. Accelerating pipeline development depends on a thorough read of the documentation, a familiarity with your scripting language’s requests library and patience dealing with pesky data structures. If you think, during this time, engineers are heads-down, you may have watched The Social Network too many times; personally, I like a little external stimuli while coding, which is how I ended up building a full dashboard during another American pastime–a baseball game. My secret? Distilling data with clean views, which I recommend over bloated source tables for both aesthetic and performance reasons. Even optimizations like views have their limitations, leading to optimization ceilings. The best way to break through, aside from stubbornness, is a combination of incremental problem-solving and “big picture” data modeling to reassess resources and attack the problem completely. Since I don’t want you to have to work any harder today, here are the embedded links as text:
If you’re celebrating America today, happy 4th! Thanks for ingesting, -Zach |
Pipeline To DE
Top data engineering writer on Medium & Senior Data Engineer in media; I use my skills as a former journalist to demystify data science/programming concepts so beginners to professionals can target, land and excel in data-driven roles.
Extract. Transform. Read. A newsletter from Pipeline: Your Data Engineering Resource Hi past, present or future data professional! Somewhere along your professional development journey someone lied to you. They told you to crank out resumes because no one reads cover letters. This couldn’t be further from the truth as 87% of hiring managers read cover letters. Such a high read rate represents a compelling opportunity to sell your data skills and showcase a bit of personality. The problem?...
The Latest From Pipeline: Your Data Engineering Resource Hi past, present or future data professional! I hope you’ll indulge my sharing of an important career milestone; unfortunately, I’m not retiring with a gold Rolex snug on my wrist. Instead, this week marks 3 years in data engineering. I’m excited about this work-iversary because it marks a period of time in which I’ve been working in data engineering longer than my prior non-data job. If you’ve read my work you know I’ve taken an...
Extract. Transform. Read. A newsletter from Pipeline: Your Data Engineering Resource Hi past, present or future data professional! Browsing through files recently, I found 100+ old resumes I used to apply for data jobs in 2021. While data science is sold as a “good career”, the truth is it’s always been tough to break in. Those looking for jobs need to do more than ever to distinguish themselves. For anyone looking for a job you may have been taught to network with recruiters and hiring...